There is for example a method of recognizing a pointer position touched by a pointer such as a finger of a user or a pointer device used by a user on the surface of a table, a wall, or the like present within a real space, a movement trajectory operation when the pointer moves while touching the surface, or the like from a camera or the like. Conventionally, the recognition of the touch operation of the above-described pointer and the tracking of the touch position are performed separately by respective different processing modules.
As the recognition of the touch operation of the pointer directed by the user, for example, there is a touch recognition method using dynamic Bayesian networks (DBNs). The DBN is a method for combining a plurality of kinds of sensor data and estimating a state. In addition, as a method for tracking the touch position, there is a tracking method using multiple hypothesis tracking (MHT), for example. The MHT enables robust tracking even when a plurality of pointers intersect each other or observation data is lost.